Layer 3 Hardware Offloading (L3HW, otherwise known as IP switching or HW routing) allows to offload some router features onto the switch chip. This allows reaching wire speeds when routing packets, which simply would not be possible with the CPU.聽

To enable Layer 3 Hardware Offloading, set l3-hw-offloading=yes for the switch:

Layer 3 Hardware Offloading can be configured for each physical switch port. For example:

Note that l3hw settings for switch and ports are different:

To enable full hardware routing, enable l3hw on all switch ports:

To make all packets go through the CPU first, and offload only the Fasttrack connections, disable l3hw on all ports but keep it enabled on the switch chip itself:

Packets get routed by the hardware only if both source and destination ports have l3-hw-offloading=yes. If at least one of them has l3-hw-offloading=no, packets will go through the CPU/Firewall while offloading only the Fasttrack connections.

The next example enables hardware routing on all ports but the upstream port (sfp-sfpplus16). Packets going to/from sfp-sfpplus16 will enter the CPU and, therefore, subject to Firewall/NAT processing.

The existing connections may be unaffected by the l3-hw-offloading setting change.

The L3HW Settings menu has been introduced in RouterOS version 7.6.

Sub-menu:聽/interface ethernet switch l3hw-settings

Property

Description

Read-Only Properties

Property

Description

This menu allows tweaking l3hw settings for specific use cases.

It is NOT recommended to change the advanced L3HW settings unless instructed by MikroTik Support or MikroTik Certified Routing Engineer. Applying incorrect settings may break the L3HW operation.

Sub-menu:聽/interface ethernet switch l3hw-settings advanced

Property

Description

The switch driver stops route indexing when聽route-queue-size聽(see聽#monitor) exceeds this value. Lowering this value leads to faster route processing but increases the lag between a route's appearance in RouterOS and hardware memory.

Setting聽route-queue-limit-high=0聽disables route indexing when there are any routes in the processing queue -聽 the most efficient CPU usage but the longest delay before hardware offloading. Useful when there are static routes only. Not recommended together with routing protocols (such as BGP or OSPF) when there are frequent routing table changes.

Re-enable route indexing when聽route-queue-size聽drops down to this value. Must not exceed the high limit.

Setting聽route-queue-limit-low=0聽tells the switch driver to process all pending routes before the next hw-offloading attempt. While this is the desired behavior, it may completely block the hw-offloading under a constant BGP feed.

Reset the Shortest HW Prefix (see聽ipv4-shortest-hw-prefix聽/聽ipv6-shortest-hw-prefix聽in聽#monitor) and try the full route table offloading after this amount of changes in the routing table. At a partial offload, when the entire routing table does not fit into the hardware memory and shorter prefixes are redirected to the CPU, there is no need to try offloading route prefixes shorter than SHWP since those will get redirected to the CPU anyway, theoretically. However, significant changes to the routing table may lead to a different index layout and, therefore, a different amount of routes that can be hw-offloaded. That's why it is recommended to do the full table re-indexing occasionally.

Lowering this value may allow more routes to be hw-offloaded but increases CPU usage and vice-versa. Setting聽shwp-reset-counter=0聽always does full re-indexing after each routing table change.

This setting is used only during Partial Offloading and has no effect when聽ipv4-shortest-hw-prefix=0聽(and ipv6, respectively).

The minimum number of routes for incremental adding in Partial Offloading. Depending on the switch chip model, routes are offloaded either as-is (each routing entry in RouterOS corresponds to an entry in the hardware memory) or getting indexed, and the index entries are the ones that are written into the hardware memory. This setting is used only for the latter during Partial Offloading.

Depending on index fragmentation, a single IPv4 route addition can occupy from -3 to +6 LPM blocks of HW memory (some route addition may lower the amount of required HW memory thanks to index defragmentation). Hence, it is impossible to predict the exact number of routes that may fit in the hardware memory. The switch driver uses a binary split algorithm to find the maximum number of routes that fit in the hardware.

Let's imagine 128k routes, all of them not fitting into the hardware memory. The algorithm halves the number and tries offloading 64k routes. Let's say offloading succeeded. In the next iteration, the algorithm picks 96k, let's say it fails; then 80k - fails again, 72k - succeeds, 76k, etc. until the difference between succeeded and failed numbers drops below the聽partial-offload-chunk聽value.

Lowering the聽partial-offload-chunk聽value increases the number of hw-offloaded routes but also raises CPU usage and vice-versa.

The minimum delay between route processing and its offloading. The delay allows processing more routes together and offloading them at once, saving CPU usage. It also makes offloading the entire routing table faster by reducing the per-route processing work. On the other hand, it slows down offloading of an individual route.

If an additional route is received during the delay, the latter resets to the聽route-index-delay-min聽value.聽Adding more and more routes within the delay keeps resetting the timer until the聽route-index-delay-max聽is reached.

The maximum delay between route processing and its offloading. When the maximum delay is reached, the processed routes get offloaded despite more routes pending. However,聽route-queue-limit-high聽has higher priority than this, meaning that the indexing/offloading gets paused anyway when a certain queue size is reached.

Neighbor (host) keepalive interval. When a host (IP neighbor) gets hw-offloaded, all traffic from/to it is routed by the switch chip, and RouterOS may think the neighbor is inactive and delete it. To prevent that, the switch driver must keep the offloaded neighbors alive by sending periodical refreshes to RouterOS.

Unfortunately, switch chips do not provide per-neighbor stats. Hence, the only way to check if the offloaded host is still active is by sending occasional ARP (IPv4) / Neighbor Discovery (IPv6) requests to the connected network. Increasing the value lowers the broadcast traffic but may leave inactive hosts in hardware memory for longer.

Neighbor discovery is triggered within the neighbor keepalive work. Hence, the discovery time is rounded up to the next keepalive session. Choose a value for聽neigh-discovery-interval聽not dividable by聽neigh-keepalive-interval聽to send ARP/ND requests in various sessions, preventing broadcast bursts.

The maximum number of ARP/ND requests that can be sent at once.

The delay between ARP/ND subsequent bursts if the number of requests exceeds聽neigh-discovery-burst-limit.

Some settings only apply to certain switch models.

The L3HW Monitor feature has been introduced in RouterOS version 7.10. It allows monitoring of switch chip and driver stats related to L3HW.聽

Stats

Property

Description

1聽IPv6 stats appear only when IPv6 hardware routing is enabled (ipv6-hw=yes)

2聽FastTrack stats appear only when hardware offloading of FastTrack connections is enabled (fasttrack-hw=yes)

An enhanced version of Monitor with extra telemetry data for advanced users. Advanced Monitor contains all data from the basic monitor聽plus the fields listed below.

Stats

Property

Description

It is impossible to use interface lists directly to control l3-hw-offloading because an interface list may contain virtual interfaces (such as VLAN) while the l3-hw-offloading setting must be applied to physical switch ports only. For example, if there are two VLAN interfaces (vlan20 and vlan30) running on the same switch port (trunk port), it is impossible to enable hardware routing on vlan20 but keep it disabled on vlan30.

However, an interface list may be used as a port selector. The following example demonstrates how to enable hardware routing on LAN ports (ports that belong to the "LAN" interface list) and disable it on WAN ports:

Please take into account that since interface lists are not used directly in the hardware routing control, modifying the interface list also does not automatically reflect into l3hw changes. For instance, adding a switch port to the "LAN" interface list does not automatically enable l3-hw-offloading on that. The user has to rerun the above script to apply the changes.

The hardware supports up to 8 MTU profiles, meaning that the user can set up to 8 different MTU values for interfaces: the default 1500 + seven custom ones.

Layer 3 hardware processing lies on top of Layer 2 hardware processing. Therefore, L3HW offloading requires L2HW offloading on the underlying interfaces. The latter is enabled by default, but there are some exceptions. For example, CRS3xx devices support only one hardware bridge. If there are multiple bridges, others are processed by the CPU and are not subject to L3HW.聽

Another example is ACL rules. If a rule redirects traffic to the CPU for software processing, then hardware routing (L3HW) is not triggered:

To make sure that Layer 3 is in sync with Layer 2 on both the software and hardware sides, we recommend disabling L3HW while configuring Layer 2 features. The recommendation applies to the following configuration:

In short, disable l3-hw-offloading while making changes under /interface/bridge/ and /interface/vlan/:

There is a limitation for MAC telnet and RoMON when L3HW offloading is enabled on 98DX8xxx, 98DX4xxx or 98DX325x switch chips. Packets from these protocols are dropped and do not reach the CPU, thus access to the device will fail.

If MAC telnet or RoMON are desired in combination with L3HW, certain ACL rules can be created to force these packets to the CPU.

For example, if MAC telnet access on sfp-sfpplus1 and sfp-sfpplus2 is needed, you will need to add this ACL rule. It is possible to select even more interfaces with the ports setting.

For example, if RoMON access on sfp-sfpplus2 is needed, you will need to add this ACL rule.

Since L3HW depends on L2HW, and L2HW is the one that does VLAN processing, Inter-VLAN hardware routing requires a hardware bridge underneath. Even if a particular VLAN has only one tagged port member, the latter must be a bridge member. Do not assign a VLAN interface directly on a switch port! Otherwise, L3HW offloading fails and the traffic will get processed by the CPU:

/interface/vlan add interface=ether2 name=vlan20 vlan-id=20

Assign VLAN interface to the bridge instead. This way, VLAN configuration gets offloaded to the hardware, and, with L3HW enabled, the traffic is subject to inter-VLAN hardware routing.

Marvell Prestera DX2000 and DX3000 switch chips have a hardware limitation that allows configuring only the last (least significant) octet of the MAC address for each interface. The other five (most significant) octets are configurated globally and, therefore, must be equal for all interfaces (switch ports, bridge, VLANs). In other words, the MAC addresses must be in the format "XX:XX:XX:XX:XX:??", where:

This requirement applies only to Layer 3 (routing). Layer 2 (bridging) does not use the switch's ethernet addresses. Moreover, it does not apply to bridge ports because they use the bridge's MAC address.

The requirement for common five octets applies to:

By default, all the routes are participating to be hardware candidate routes. To further fine-tune which traffic to offload, there is an option for each route to disable/enable聽suppress-hw-offload.聽

For example, if we know that majority of traffic flows to the network where servers are located, we can enable offloading only to that specific destination:

Now only the route to 192.168.3.0/24 has H-flag, indicating that it will be the only one eligible to be selected for HW offloading:

H-flag does not indicate that route is actually HW offloaded, it indicates only that route can be selected to be HW offloaded.

For dynamic routing protocols like OSFP and BGP, it is possible to suppress HW offloading using聽routing filters. For example, to suppress HW offloading on all OSFP instance routes, use "suppress-hw-offload yes" property:

Firewall filter rules have hw-offload option for Fasttrack, allowing fine-tuning connection offloading. Since the hardware memory for Fasttrack connections is very limited, we can choose what type of connections to offload and, therefore, benefit from near-the-wire-speed traffic. The next example offloads only TCP connections while UDP packets are routed via the CPU and do not occupy HW memory:

While connection tracking and stateful firewalling can be performed only by the CPU, the hardware can perform stateless firewalling via聽switch rules (ACL). The next example prevents (on a hardware level) accessing a MySQL server from the ether1, and redirects to the CPU/Firewall packets from ether2 and ether3:

Some firewall rules may be implemented both via聽switch rules (ACL) and CPU聽Firewall Filter聽+ Fasttrack HW Offloading. Both options grant near-the-wire-speed performance. So the question is which one to use?

First,聽not all devices support Fasttrack HW Offloading. And without HW offloading, Firewall Filter uses only software routing, which is dramatically slower than its hardware counterpart. Second, even if Fasttrack HW Offloading is an option, a rule of thumb is:

Always use Switch Rules (ACL), if possible.

Switch rules share the hardware memory with Fastrack connections. However, hardware resources are allocated for each Fasttrack connection while a single ACL rule can match multiple connections. For example, if you have a guest WiFi network connected to sfp-sfpplus1 VLAN 10 and you don't want it to access your internal network, simply create an ACL rule:

The matched packets will be dropped on the hardware level. It is much better than letting聽all聽guest packets to the CPU for Firewall filtering.

Of course, ACL rules cannot match everything. For instance, ACL rules cannot filter connection states: accept established, drop others. That is where Fasttrack HW Offloading gets into action - redirect the packets to the CPU by default for firewall filtering, then offload the established Fasttrack connections. However, disabling聽l3-hw-offloading聽for the entire switch port is not the only option.

Define ACL rules with redirect-to-cpu=yes instead of setting l3-hw-offloading=no of the switch port for narrowing down the traffic that goes to the CPU.

This example demonstrates how to benefit from near-to-wire-speed inter-VLAN routing while keeping Firewall and NAT running on the upstream port. Moreover, Fasttrack connections to the upstream port get offloaded to hardware as well, boosting the traffic speed close to wire-level. Inter-VLAN traffic is fully routed by the hardware, not entering the CPU/Firewall, and, therefore, not occupying the hardware memory of Fasttrack connections.

We use the CRS317-1G-16S+ model with the following setup:

Setup interface lists for easy access:

Routing requires dedicated VLAN interfaces. For standard L2 VLAN bridging (without inter-VLAN routing), the next step can be omitted.

Configure management and upstream ports, a basic firewall, NAT, and enable hardware offloading of Fasttrack connections:

At this moment, all routing still is performed by the CPU. Enable hardware routing on the switch chip:

Results:

Below are typical user errors of configuring Layer 3 Hardware Offloading.

VLAN interface must be set on the bridge due to Layer 2 Dependency. Otherwise, L3HW will not work. The correct configuration is:

For Inter-VLAN routing, the bridge interface itself needs to be added to the tagged members of the given VLANs. In the next example, Inter-VLAN routing works between VLAN 10 and 11, but packets are NOT routed to VLAN 20.聽

The above example does not always mean an error. Sometimes, you may want the device to act as a simple L2 switch in some/all VLANs. Just make sure you set such behavior on purpose, not due to a mistake.

The devices support only one hardware bridge. If there are multiple bridges created, only one gets hardware offloading. While for L2 that means software forwarding for other bridges, in the case of L3HW, multiple bridges may lead to undefined behavior.

Instead of creating multiple bridges, create one and segregate L2 networks with VLAN filtering.

Some devices have two switch chips or the management port directly connected to the CPU. For example,聽CRS312-4C+8XG聽has an聽ether9聽port connected to a separate switch chip. Trying to add this port to a bridge or involve it in the L3HW setup leads to unexpected results. Leave the management port for management!

Since Fasttrack HW Offloading offers near-the-wire-speed performance at zero configuration overhead, the users tempt to use it as the default solution. However, the number of HW Fasttrack connections is very limited, leaving the other traffic for the CPU. Try using the hardware routing as much as possible, reduce the CPU traffic to the minimum via switch ACL rules, and then fine-tune which Fasttrack connections to offload with firewall filter rules.

Using certain configuration (e.g. enabling bridge "use-ip-firewall" setting, creating bridge nat/filter rules) or running specific features like sniffer or torch can disable RouterOS FastPath, which will affect the ability to properly FastTrack and HW offload connections. If HW offloaded Fasttrack is required, make sure that there are no settings that disable the FastPath and verify that connections are getting the "H" flag or use the L3HW monitor command to see the amount of HW offloaded connections.

This works only for directly connected networks. Since HW does not know how to send ARP requests,CPU sends an ARP request and waits for a reply to find out a DST MAC address on the first received packet of the connection that matches a DST IP address.After DST MAC is determined, HW entry is added and all further packets will be processed by the switch chip.

Only the devices listed in the table below support L3 HW Offloading.

Only the devices listed in the table below support L3 HW Offloading.

The devices below are based on Marvell 98DX224S, 98DX226S, or 98DX3236 switch chip models. These devices do not support Fasttrack or NAT connection offloading.

The 98DX3255 and 98DX3257 models are exceptions, which have a feature set of the DX8000 rather than the DX3000 series.

1 Since the total amount of routes that can be offloaded is limited, prefixes with higher netmask are preferred to be forwarded by hardware (e.g., /32, /30, /29, etc.), any other prefixes that do not fit in the HW table will be processed by the CPU. Directly connected hosts are offloaded as /32 (IPv4) or /128 (IPv6) route prefixes. The number of hosts is also limited by max-neighbor-entries in IP Settings / IPv6 Settings.

2聽IPv4 and IPv6 routing tables share the same hardware memory.

3聽If a route has more paths than the hardware ECMP limit (X), only the first X paths get offloaded.

The devices below are based on Marvell 98DX8xxx, 98DX4xxx switch chips, or 98DX325x聽model.

1聽Depends on the complexity of the routing table. Whole-byte IP prefixes (/8, /16, /24, etc.) occupy less HW space than others (e.g., /22). Starting with聽RouterOS v7.3, when the Routing HW table gets full, only routes with longer subnet prefixes are offloaded (/30, /29, /28, etc.) while the CPU processes the shorter prefixes. In RouterOS v7.2 and before, Routing HW memory overflow led to undefined behavior. Users can fine-tune what routes to offload via routing filters (for dynamic routes) or suppressing hardware offload of static routes. IPv4 and IPv6 routing tables share the same hardware memory.

2聽When the HW limit of Fasttrack or NAT entries is reached, other connections will fall back to the CPU. MikroTik's smart connection offload algorithm ensures that the connections with the most traffic are offloaded to the hardware.

3聽Fasttrack connections share the same HW memory with ACL rules. Depending on the complexity, one ACL rule may occupy the memory of 3-6 Fasttrack connections.

4聽MPLS shares the HW memory with Fasttrack connections. Moreover, enabling MPLS requires the allocation of the entire memory region, which could otherwise store up to 768 (0.75K) Fasttrack connections. The same applies to Bridge Port Extender. However, MPLS and BPE may use the same memory region, so enabling them both doesn't double the limitation of Fasttrack connections.

5聽If a Fasttrack connection requires Network Address Translation, a hardware NAT entry is created. The hardware supports both SRCNAT and DSTNAT.

6聽The switch chip has a feature set of the DX8000 series.

7聽DX4000/DX8000 switch chips store directly connected hosts, IPv4 /32, and IPv6 /128 route entries in the FDB table rather than the routing table. The HW memory is shared between regular FDB L2 entries (MAC), IPv4, and IPv6 addresses. The number of hosts is also limited by max-neighbor-entries in IP Settings / IPv6 Settings.

8聽IPv4 and IPv6 routing tables share the same hardware memory.

1聽The switch chip has a feature set of the DX8000 series.